A Study on Distributed Frequent Co-occurrence Patterns Algorithms across Multiple Data Streams

With the era of big data coming, the data streams are fast, continuous, and unbounded. The real-time requirements of the data streams processing results are very high. A large number of researches have been on Frequent Co-occurrence Patterns across multiple data streams. But those algorithms are centralized, which is worked on a single compute node. The memory of a single compute node and CPU calculation can be limited, which is difficult to deal with the increasing data streams. Using the distributed server cluster is an effective way. However, the centralized algorithm cannot be directly deployed to distributed server cluster. This paper designs a Distributed Frequent Co-occurrence Pattern across multiple data streams to solve these problems. Through a lot of experiments to evaluate it, the algorithm can detect all the objects that meet the conditions in real time, and have good scalability. In order to save memory, this paper also improves the algorithm, and proposes Modified Distributed Frequent Co-occurrence Pattern based on P-condition deletion strategy. The improved algorithm can delete element combinations which can not constitute Frequent Co-occurrence Patterns in the initial stage, so as to effectively save memory.